Skip to content

ansible: Drop unnecessary updates/upgrades - #2353

Open
mmlb wants to merge 8 commits into
developfrom
manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade
Open

ansible: Drop unnecessary updates/upgrades#2353
mmlb wants to merge 8 commits into
developfrom
manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade

Conversation

@mmlb

@mmlb mmlb commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

What kind of change does this PR introduce?

Maintenance

What is the current behavior?

We run apt-get update almost every time we install a package in ansible and we run an apt-get upgrade in the middle of the AMI creation process. Both update and upgrade are already being done before and after ansible runs which is better. Have a good known state early and update/upgrade at the end to ensure we have latest versions. Could have been done in ansible too but since we already do pre/post ansible apt calls I decided to just leave them there.

What is the new behavior?

We don't waste time with unnecessary apt updates during ansible which could slow down the run and/or introduce flakiness if the remote is being updated too. No need to do the upgrade either since it was already being done post ansible.

@mmlb
mmlb requested review from a team as code owners August 10, 2026 17:01
@mmlb
mmlb force-pushed the manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade branch 3 times, most recently from 5444465 to f224629 Compare August 10, 2026 19:57
mmlb and others added 8 commits August 24, 2026 10:40
Having swap setup in stage1 uses up 10% of the disk while we need as
much space as possible (for fetching, building, etc). We are extremely
sensitive to cache clean up ordering so that we don't go over, but thats
a big pain that can be allievated by moving swap to the end of the run.
We run stage2 on beefy machines that don't need swap so lets use the
space better.
It turns out that most of the code deleted here was actually not doing
anything useful. For example, switch_mirror modifies
/etc/apt/sources.list in place but the file doesn't have any mirrors
configured there so it's really a no-op! Here's the contents from an
instance I just fired up[^1]:

    ubuntu@ip-172-31-26-227:~$ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/*
    ==> /etc/apt/sources.list <==
    # Ubuntu sources have moved to the /etc/apt/sources.list.d/ubuntu.sources
    # file, which uses the deb822 format. Use deb822-formatted .sources files
    # to manage package sources in the /etc/apt/sources.list.d/ directory.
    # See the sources.list(5) manual page for details.

    ==> /etc/apt/sources.list.d/ubuntu.sources <==
    ## Note, this file is written by cloud-init on first boot of an instance
    ## modifications made here will not survive a re-bundle.
    --- 8< ---
    Types: deb
    URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
    Suites: noble noble-updates noble-backports
    Components: main universe restricted multiverse
    Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
    Types: deb
    URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
    Suites: noble noble-updates noble-backports
    Components: main universe restricted multiverse
    Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

    ## Ubuntu security updates. Aside from URIs and Suites,
    ## this should mirror your choices in the previous section.
    Types: deb
    URIs: http://security.ubuntu.com/ubuntu
    Suites: noble-security
    Components: main universe restricted multiverse
    Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg

We get fallback handling by apt itself by adding multiple mirrors in
URIs, apt tries first (the regional) and falls back to global ubuntu
repos if there's an issue. We also setup a temporary apt config that
forces the APT_OPTIONS for all apt calls so we don't need to do it in
every call site.

I ended up dropping the in-repo sources file since they are strictly
worse than what we get from AWS in the build. There's no change in
suites or components between old and new, just that we get ubuntu
upstream as a fallback. Well technically there's a slight difference
since we are basing off of cloud-init generated files and they can
theoretically change under us but I'll gamble that it'll be fine or
better off. Besides, one day we'll be on NixOS as the ultimate "make
sure we know everything in the instance" ;).

I also got rid of the `add-apt-repository --yes universe` because
universe is already enabled.

[^1]: AMI=ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20260604
We are wasting a bunch of time and brain power keeping track of package
installs, so lets minimize them while keeping the grouping/messages
intact.

I did move the package installs from setup_apparmor && setup_grub into
update_install_packages because why not just install everything in one
place and leave the _setup_ to somewhere else. I moved them out of
because package installation should all be done early and ASAP IMO so it
can fail fast if its going to fail.

Also I moved ec2-hibinit-agent, ec2-instance-connect, hibagent here
since this is only run for AMIs and the comment isn't currently
accurate. Either through `export DEBIAN_FRONTEND=noninteractive`,
`disable_services` or maybe it was treated as a bug and is now fixed in
the package I'm not sure but I ran this out of AWS and it was fine.
Lets be a little cleaner about setup/update/cleanup, also drop the
wrapper functions to plain old apt-get calls since they no longer do
anything useful.

I created setup_apt because I want to move the Install-Recommends=False
into it later and configure all apt calls to use it seamlessly, this way
we will have a better/more visible list of packages being installed.
No yum handling necessary and commented out code should be deleted.
There's no point run updates/upgrades unnecessarily while also not
ensuring that we start with latest and end with latest. So I've dropped
any unecessary updates and upgrades from middle of runs. Now we update
and upgrade after start up ({surrogate,qemu}-bootstrap-nix, chroot was
already good) and then once at the end (qemu-bootstrap-nix and
nix-provision). Ansible's apt_repository module does update when
necessary and has not changed.
dist-upgrade ensures new packages are installed and unnecessary are
removed while upgrade only upgrades packages. Using dist-upgrade gets us
closer to inteded state according to metadata files.
@mmlb
mmlb force-pushed the manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade branch from f224629 to ebe2013 Compare August 24, 2026 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant